<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>
  Class: Upton::Scraper
  
    &mdash; Documentation by YARD 0.8.7
  
</title>

  <link rel="stylesheet" href="../css/style.css" type="text/css" charset="utf-8" />

  <link rel="stylesheet" href="../css/common.css" type="text/css" charset="utf-8" />

<script type="text/javascript" charset="utf-8">
  hasFrames = window.top.frames.main ? true : false;
  relpath = '../';
  framesUrl = "../frames.html#!" + escape(window.location.href);
</script>


  <script type="text/javascript" charset="utf-8" src="../js/jquery.js"></script>

  <script type="text/javascript" charset="utf-8" src="../js/app.js"></script>


  </head>
  <body>
    <div id="header">
      <div id="menu">
  
    <a href="../_index.html">Index (S)</a> &raquo;
    <span class='title'><span class='object_link'><a href="../Upton.html" title="Upton (module)">Upton</a></span></span>
     &raquo; 
    <span class="title">Scraper</span>
  

  <div class="noframes"><span class="title">(</span><a href="." target="_top">no frames</a><span class="title">)</span></div>
</div>

      <div id="search">
  
    <a class="full_list_link" id="class_list_link"
        href="../class_list.html">
      Class List
    </a>
  
    <a class="full_list_link" id="method_list_link"
        href="../method_list.html">
      Method List
    </a>
  
    <a class="full_list_link" id="file_list_link"
        href="../file_list.html">
      File List
    </a>
  
</div>
      <div class="clear"></div>
    </div>

    <iframe id="search_frame"></iframe>

    <div id="content"><h1>Class: Upton::Scraper
  
  
  
</h1>

<dl class="box">
  
    <dt class="r1">Inherits:</dt>
    <dd class="r1">
      <span class="inheritName">Object</span>
      
        <ul class="fullTree">
          <li>Object</li>
          
            <li class="next">Upton::Scraper</li>
          
        </ul>
        <a href="#" class="inheritanceTree">show all</a>
      
      </dd>
    
  
  
    
  
    
  
  
  
    <dt class="r2 last">Defined in:</dt>
    <dd class="r2 last">lib/upton.rb</dd>
  
</dl>
<div class="clear"></div>

<h2>Overview</h2><div class="docstring">
  <div class="discussion">
    
<p><strong>Upton</strong> is a framework for easy web-scraping with a useful
debug mode  that doesn&#39;t hammer your target&#39;s servers. It does the
repetitive parts of  writing scrapers, so you only have to write the unique
parts for each site.</p>

<p>Upton operates on the theory that, for most scraping projects, you need to
scrape two types of pages:</p>
<ol><li>
<p>Index pages, which list instance pages. For example, a job search</p>

<pre class="code ruby"><code class="ruby">site&#39;s search page or a newspaper&#39;s homepage.</code></pre>
</li><li>
<p>Instance pages, which represent the goal of your scraping, e.g.</p>

<pre class="code ruby"><code class="ruby">job listings or news articles.</code></pre>
</li></ol>

<p>Upton::Scraper can be used as-is for basic use-cases by:</p>
<ol><li>
<p>specifying the pages to be scraped in `new` as an index page</p>

<pre class="code ruby"><code class="ruby">or as an Array of URLs.</code></pre>
</li><li>
<p>supplying a block to `scrape` or `scrape_to_csv` or using a pre-build</p>

<pre class="code ruby"><code class="ruby">block from Upton::Utils.</code></pre>
</li></ol>

<p>For more complicated cases; subclass Upton::Scraper</p>

<pre class="code ruby"><code class="ruby">e.g. +MyScraper &lt; Upton::Scraper+ and override various methods.</code></pre>


  </div>
</div>
<div class="tags">
  

</div>



  <h2>Instance Attribute Summary <small>(<a href="#" class="summary_toggle">collapse</a>)</small></h2>
  <ul class="summary">
    
      <li class="public ">
  <span class="summary_signature">
    
      <a href="#debug-instance_method" title="#debug (instance method)">- (Object) <strong>debug</strong> </a>
    

    
  </span>
  
  
  
    
    
  
  
  
  
  

  
    <span class="summary_desc"><div class='inline'>
<p>Returns the value of attribute debug.</p>
</div></span>
  
</li>

    
      <li class="public ">
  <span class="summary_signature">
    
      <a href="#paginated-instance_method" title="#paginated (instance method)">- (Object) <strong>paginated</strong> </a>
    

    
  </span>
  
  
  
    
    
  
  
  
  
  

  
    <span class="summary_desc"><div class='inline'>
<p>Returns the value of attribute paginated.</p>
</div></span>
  
</li>

    
      <li class="public ">
  <span class="summary_signature">
    
      <a href="#pagination_max_pages-instance_method" title="#pagination_max_pages (instance method)">- (Object) <strong>pagination_max_pages</strong> </a>
    

    
  </span>
  
  
  
    
    
  
  
  
  
  

  
    <span class="summary_desc"><div class='inline'>
<p>Returns the value of attribute pagination_max_pages.</p>
</div></span>
  
</li>

    
      <li class="public ">
  <span class="summary_signature">
    
      <a href="#pagination_param-instance_method" title="#pagination_param (instance method)">- (Object) <strong>pagination_param</strong> </a>
    

    
  </span>
  
  
  
    
    
  
  
  
  
  

  
    <span class="summary_desc"><div class='inline'>
<p>Returns the value of attribute pagination_param.</p>
</div></span>
  
</li>

    
      <li class="public ">
  <span class="summary_signature">
    
      <a href="#sleep_time_between_requests-instance_method" title="#sleep_time_between_requests (instance method)">- (Object) <strong>sleep_time_between_requests</strong> </a>
    

    
  </span>
  
  
  
    
    
  
  
  
  
  

  
    <span class="summary_desc"><div class='inline'>
<p>Returns the value of attribute sleep_time_between_requests.</p>
</div></span>
  
</li>

    
      <li class="public ">
  <span class="summary_signature">
    
      <a href="#stash_folder-instance_method" title="#stash_folder (instance method)">- (Object) <strong>stash_folder</strong> </a>
    

    
  </span>
  
  
  
    
    
  
  
  
  
  

  
    <span class="summary_desc"><div class='inline'>
<p>Returns the value of attribute stash_folder.</p>
</div></span>
  
</li>

    
      <li class="public ">
  <span class="summary_signature">
    
      <a href="#url_array-instance_method" title="#url_array (instance method)">- (Object) <strong>url_array</strong> </a>
    

    
  </span>
  
  
  
    
    
  
  
  
  
  

  
    <span class="summary_desc"><div class='inline'>
<p>Returns the value of attribute url_array.</p>
</div></span>
  
</li>

    
      <li class="public ">
  <span class="summary_signature">
    
      <a href="#verbose-instance_method" title="#verbose (instance method)">- (Object) <strong>verbose</strong> </a>
    

    
  </span>
  
  
  
    
    
  
  
  
  
  

  
    <span class="summary_desc"><div class='inline'>
<p>Returns the value of attribute verbose.</p>
</div></span>
  
</li>

    
  </ul>




  
    <h2>
      Instance Method Summary
      <small>(<a href="#" class="summary_toggle">collapse</a>)</small>
    </h2>

    <ul class="summary">
      
        <li class="public ">
  <span class="summary_signature">
    
      <a href="#initialize-instance_method" title="#initialize (instance method)">- (Scraper) <strong>initialize</strong>(index_url_or_array, selector = &quot;&quot;, selector_method = :deprecated) </a>
    

    
  </span>
  
  
    <span class="note title constructor">constructor</span>
  
  
  
  
  
  

  
    <span class="summary_desc"><div class='inline'>
<p>the selector_method parameter is unneeded, as Nokogiri provides the 
#search method, which picks a selector depending on whether  the String
passed is of CSS/XPath notation.</p>
</div></span>
  
</li>

      
        <li class="public ">
  <span class="summary_signature">
    
      <a href="#next_index_page_url-instance_method" title="#next_index_page_url (instance method)">- (Object) <strong>next_index_page_url</strong>(url, pagination_index) </a>
    

    
  </span>
  
  
  
  
  
  
  

  
    <span class="summary_desc"><div class='inline'>
<p>Return the next URL to scrape, given the current URL and its index.</p>
</div></span>
  
</li>

      
        <li class="public ">
  <span class="summary_signature">
    
      <a href="#next_instance_page_url-instance_method" title="#next_instance_page_url (instance method)">- (Object) <strong>next_instance_page_url</strong>(url, pagination_index) </a>
    

    
  </span>
  
  
  
  
  
  
  

  
    <span class="summary_desc"><div class='inline'>
<p>If instance pages are paginated, you must override  this method to return
the next URL, given the current URL and its index.</p>
</div></span>
  
</li>

      
        <li class="public ">
  <span class="summary_signature">
    
      <a href="#scrape-instance_method" title="#scrape (instance method)">- (Object) <strong>scrape</strong>(&amp;blk) </a>
    

    
  </span>
  
  
  
  
  
  
  

  
    <span class="summary_desc"><div class='inline'>
<p>This is the main user-facing method for a basic scraper.</p>
</div></span>
  
</li>

      
        <li class="public ">
  <span class="summary_signature">
    
      <a href="#scrape_to_csv-instance_method" title="#scrape_to_csv (instance method)">- (Object) <strong>scrape_to_csv</strong>(filename, &amp;blk) </a>
    

    
  </span>
  
  
  
  
  
  
  

  
    <span class="summary_desc"><div class='inline'>
<p>Writes the scraped result to a CSV at the given filename.</p>
</div></span>
  
</li>

      
        <li class="public ">
  <span class="summary_signature">
    
      <a href="#scrape_to_tsv-instance_method" title="#scrape_to_tsv (instance method)">- (Object) <strong>scrape_to_tsv</strong>(filename, &amp;blk) </a>
    

    
  </span>
  
  
  
  
  
  
  

  
    <span class="summary_desc"><div class='inline'></div></span>
  
</li>

      
    </ul>
  

<div id="constructor_details" class="method_details_list">
  <h2>Constructor Details</h2>
  
    <div class="method_details first">
  <h3 class="signature first" id="initialize-instance_method">
  
    - (<tt><span class='object_link'><a href="" title="Upton::Scraper (class)">Scraper</a></span></tt>) <strong>initialize</strong>(index_url_or_array, selector = &quot;&quot;, selector_method = :deprecated) 
  

  

  
</h3><div class="docstring">
  <div class="discussion">
    
<p>the selector_method parameter is unneeded, as Nokogiri provides the</p>

<pre class="code ruby"><code class="ruby">#search method, which picks a selector depending on whether
the String passed is of CSS/XPath notation</code></pre>


  </div>
</div>
<div class="tags">
  

</div><table class="source_code">
  <tr>
    <td>
      <pre class="lines">


71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117</pre>
    </td>
    <td>
      <pre class="code"><span class="info file"># File 'lib/upton.rb', line 71</span>

<span class='kw'>def</span> <span class='id identifier rubyid_initialize'>initialize</span><span class='lparen'>(</span><span class='id identifier rubyid_index_url_or_array'>index_url_or_array</span><span class='comma'>,</span> <span class='id identifier rubyid_selector'>selector</span><span class='op'>=</span><span class='tstring'><span class='tstring_beg'>&quot;</span><span class='tstring_end'>&quot;</span></span><span class='comma'>,</span> <span class='id identifier rubyid_selector_method'>selector_method</span><span class='op'>=</span><span class='symbol'>:deprecated</span><span class='rparen'>)</span>
  
  <span class='comment'>#if first arg is a valid URL, do already-written stuff;
</span>  <span class='comment'>#if it&#39;s not (or if it&#39;s a list?) don&#39;t bother with get_index, etc.
</span>  <span class='comment'>#e.g. Scraper.new([&quot;http://jeremybmerrill.com&quot;])
</span>
  <span class='comment'>#TODO: rewrite this, because it&#39;s a little silly. (i.e. should be a more sensical division of how these arguments work)
</span>  <span class='kw'>if</span> <span class='id identifier rubyid_index_url_or_array'>index_url_or_array</span><span class='period'>.</span><span class='id identifier rubyid_respond_to?'>respond_to?</span> <span class='symbol'>:each_with_index</span>
    <span class='ivar'>@url_array</span> <span class='op'>=</span> <span class='id identifier rubyid_index_url_or_array'>index_url_or_array</span>
  <span class='kw'>else</span>
    <span class='ivar'>@index_url</span> <span class='op'>=</span> <span class='id identifier rubyid_index_url_or_array'>index_url_or_array</span>
    <span class='ivar'>@index_selector</span> <span class='op'>=</span> <span class='id identifier rubyid_selector'>selector</span>
  <span class='kw'>end</span>
  <span class='comment'># If true, then Upton prints information about when it gets
</span>  <span class='comment'># files from the internet and when it gets them from its stash.
</span>  <span class='ivar'>@verbose</span> <span class='op'>=</span> <span class='kw'>false</span>

  <span class='comment'># If true, then Upton fetches each instance page only once
</span>  <span class='comment'># future requests for that file are responded to with the locally stashed
</span>  <span class='comment'># version.
</span>  <span class='comment'># You may want to set @debug to false for production (but maybe not).
</span>  <span class='comment'># You can also control stashing behavior on a per-call basis with the
</span>  <span class='comment'># optional second argument to get_page, if, for instance, you want to 
</span>  <span class='comment'># stash certain instance pages, e.g. based on their modification date.
</span>  <span class='ivar'>@debug</span> <span class='op'>=</span> <span class='kw'>true</span>
  <span class='comment'># Index debug does the same, but for index pages.
</span>  <span class='ivar'>@index_debug</span> <span class='op'>=</span> <span class='kw'>false</span>

  <span class='comment'># In order to not hammer servers, Upton waits for, by default, 30  
</span>  <span class='comment'># seconds between requests to the remote server.
</span>  <span class='ivar'>@sleep_time_between_requests</span> <span class='op'>=</span> <span class='int'>30</span> <span class='comment'>#seconds
</span>
  <span class='comment'># If true, then Upton will attempt to scrape paginated index pages
</span>  <span class='ivar'>@paginated</span> <span class='op'>=</span> <span class='kw'>false</span>
  <span class='comment'># Default query string parameter used to specify the current page
</span>  <span class='ivar'>@pagination_param</span> <span class='op'>=</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>page</span><span class='tstring_end'>&#39;</span></span>
  <span class='comment'># Default number of paginated pages to scrape
</span>  <span class='ivar'>@pagination_max_pages</span> <span class='op'>=</span> <span class='int'>2</span>


  <span class='comment'># Folder name for stashes, if you want them to be stored somewhere else,
</span>  <span class='comment'># e.g. under /tmp.
</span>  <span class='ivar'>@stash_folder</span> <span class='op'>||=</span> <span class='tstring'><span class='tstring_beg'>&quot;</span><span class='tstring_content'>stashes</span><span class='tstring_end'>&quot;</span></span>
  <span class='kw'>unless</span> <span class='const'>Dir</span><span class='period'>.</span><span class='id identifier rubyid_exists?'>exists?</span><span class='lparen'>(</span><span class='ivar'>@stash_folder</span><span class='rparen'>)</span>
    <span class='const'>FileUtils</span><span class='period'>.</span><span class='id identifier rubyid_mkdir_p'>mkdir_p</span><span class='lparen'>(</span><span class='ivar'>@stash_folder</span><span class='rparen'>)</span>
  <span class='kw'>end</span>
<span class='kw'>end</span></pre>
    </td>
  </tr>
</table>
</div>
  
</div>

  <div id="instance_attr_details" class="attr_details">
    <h2>Instance Attribute Details</h2>
    
      
      <span id="debug=-instance_method"></span>
      <div class="method_details first">
  <h3 class="signature first" id="debug-instance_method">
  
    - (<tt>Object</tt>) <strong>debug</strong> 
  

  

  
</h3><div class="docstring">
  <div class="discussion">
    
<p>Returns the value of attribute debug</p>


  </div>
</div>
<div class="tags">
  

</div><table class="source_code">
  <tr>
    <td>
      <pre class="lines">


35
36
37</pre>
    </td>
    <td>
      <pre class="code"><span class="info file"># File 'lib/upton.rb', line 35</span>

<span class='kw'>def</span> <span class='id identifier rubyid_debug'>debug</span>
  <span class='ivar'>@debug</span>
<span class='kw'>end</span></pre>
    </td>
  </tr>
</table>
</div>
    
      
      <span id="paginated=-instance_method"></span>
      <div class="method_details ">
  <h3 class="signature " id="paginated-instance_method">
  
    - (<tt>Object</tt>) <strong>paginated</strong> 
  

  

  
</h3><div class="docstring">
  <div class="discussion">
    
<p>Returns the value of attribute paginated</p>


  </div>
</div>
<div class="tags">
  

</div><table class="source_code">
  <tr>
    <td>
      <pre class="lines">


35
36
37</pre>
    </td>
    <td>
      <pre class="code"><span class="info file"># File 'lib/upton.rb', line 35</span>

<span class='kw'>def</span> <span class='id identifier rubyid_paginated'>paginated</span>
  <span class='ivar'>@paginated</span>
<span class='kw'>end</span></pre>
    </td>
  </tr>
</table>
</div>
    
      
      <span id="pagination_max_pages=-instance_method"></span>
      <div class="method_details ">
  <h3 class="signature " id="pagination_max_pages-instance_method">
  
    - (<tt>Object</tt>) <strong>pagination_max_pages</strong> 
  

  

  
</h3><div class="docstring">
  <div class="discussion">
    
<p>Returns the value of attribute pagination_max_pages</p>


  </div>
</div>
<div class="tags">
  

</div><table class="source_code">
  <tr>
    <td>
      <pre class="lines">


35
36
37</pre>
    </td>
    <td>
      <pre class="code"><span class="info file"># File 'lib/upton.rb', line 35</span>

<span class='kw'>def</span> <span class='id identifier rubyid_pagination_max_pages'>pagination_max_pages</span>
  <span class='ivar'>@pagination_max_pages</span>
<span class='kw'>end</span></pre>
    </td>
  </tr>
</table>
</div>
    
      
      <span id="pagination_param=-instance_method"></span>
      <div class="method_details ">
  <h3 class="signature " id="pagination_param-instance_method">
  
    - (<tt>Object</tt>) <strong>pagination_param</strong> 
  

  

  
</h3><div class="docstring">
  <div class="discussion">
    
<p>Returns the value of attribute pagination_param</p>


  </div>
</div>
<div class="tags">
  

</div><table class="source_code">
  <tr>
    <td>
      <pre class="lines">


35
36
37</pre>
    </td>
    <td>
      <pre class="code"><span class="info file"># File 'lib/upton.rb', line 35</span>

<span class='kw'>def</span> <span class='id identifier rubyid_pagination_param'>pagination_param</span>
  <span class='ivar'>@pagination_param</span>
<span class='kw'>end</span></pre>
    </td>
  </tr>
</table>
</div>
    
      
      <span id="sleep_time_between_requests=-instance_method"></span>
      <div class="method_details ">
  <h3 class="signature " id="sleep_time_between_requests-instance_method">
  
    - (<tt>Object</tt>) <strong>sleep_time_between_requests</strong> 
  

  

  
</h3><div class="docstring">
  <div class="discussion">
    
<p>Returns the value of attribute sleep_time_between_requests</p>


  </div>
</div>
<div class="tags">
  

</div><table class="source_code">
  <tr>
    <td>
      <pre class="lines">


35
36
37</pre>
    </td>
    <td>
      <pre class="code"><span class="info file"># File 'lib/upton.rb', line 35</span>

<span class='kw'>def</span> <span class='id identifier rubyid_sleep_time_between_requests'>sleep_time_between_requests</span>
  <span class='ivar'>@sleep_time_between_requests</span>
<span class='kw'>end</span></pre>
    </td>
  </tr>
</table>
</div>
    
      
      <span id="stash_folder=-instance_method"></span>
      <div class="method_details ">
  <h3 class="signature " id="stash_folder-instance_method">
  
    - (<tt>Object</tt>) <strong>stash_folder</strong> 
  

  

  
</h3><div class="docstring">
  <div class="discussion">
    
<p>Returns the value of attribute stash_folder</p>


  </div>
</div>
<div class="tags">
  

</div><table class="source_code">
  <tr>
    <td>
      <pre class="lines">


35
36
37</pre>
    </td>
    <td>
      <pre class="code"><span class="info file"># File 'lib/upton.rb', line 35</span>

<span class='kw'>def</span> <span class='id identifier rubyid_stash_folder'>stash_folder</span>
  <span class='ivar'>@stash_folder</span>
<span class='kw'>end</span></pre>
    </td>
  </tr>
</table>
</div>
    
      
      <span id="url_array=-instance_method"></span>
      <div class="method_details ">
  <h3 class="signature " id="url_array-instance_method">
  
    - (<tt>Object</tt>) <strong>url_array</strong> 
  

  

  
</h3><div class="docstring">
  <div class="discussion">
    
<p>Returns the value of attribute url_array</p>


  </div>
</div>
<div class="tags">
  

</div><table class="source_code">
  <tr>
    <td>
      <pre class="lines">


35
36
37</pre>
    </td>
    <td>
      <pre class="code"><span class="info file"># File 'lib/upton.rb', line 35</span>

<span class='kw'>def</span> <span class='id identifier rubyid_url_array'>url_array</span>
  <span class='ivar'>@url_array</span>
<span class='kw'>end</span></pre>
    </td>
  </tr>
</table>
</div>
    
      
      <span id="verbose=-instance_method"></span>
      <div class="method_details ">
  <h3 class="signature " id="verbose-instance_method">
  
    - (<tt>Object</tt>) <strong>verbose</strong> 
  

  

  
</h3><div class="docstring">
  <div class="discussion">
    
<p>Returns the value of attribute verbose</p>


  </div>
</div>
<div class="tags">
  

</div><table class="source_code">
  <tr>
    <td>
      <pre class="lines">


35
36
37</pre>
    </td>
    <td>
      <pre class="code"><span class="info file"># File 'lib/upton.rb', line 35</span>

<span class='kw'>def</span> <span class='id identifier rubyid_verbose'>verbose</span>
  <span class='ivar'>@verbose</span>
<span class='kw'>end</span></pre>
    </td>
  </tr>
</table>
</div>
    
  </div>


  <div id="instance_method_details" class="method_details_list">
    <h2>Instance Method Details</h2>

    
      <div class="method_details first">
  <h3 class="signature first" id="next_index_page_url-instance_method">
  
    - (<tt>Object</tt>) <strong>next_index_page_url</strong>(url, pagination_index) 
  

  

  
</h3><div class="docstring">
  <div class="discussion">
    
<p>Return the next URL to scrape, given the current URL and its index.</p>

<p>Recursion stops if the fetching URL returns an empty string or an error.</p>

<p>If @paginated is not set (the default), this method returns an empty
string.</p>

<p>If @paginated is set, this method will return the next pagination URL to
scrape using @pagination_param and the pagination_index.</p>

<p>If the pagination_index is greater than @pagination_max_pages, then the
method will return an empty string.</p>

<p>Override this method to handle pagination is an alternative way e.g.
next_index_page_url(“<a
href="http://whatever.com/articles?page=1">whatever.com/articles?page=1</a>”,
2) ought to return “<a
href="http://whatever.com/articles?page=2">whatever.com/articles?page=2</a>”</p>


  </div>
</div>
<div class="tags">
  

</div><table class="source_code">
  <tr>
    <td>
      <pre class="lines">


152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167</pre>
    </td>
    <td>
      <pre class="code"><span class="info file"># File 'lib/upton.rb', line 152</span>

<span class='kw'>def</span> <span class='id identifier rubyid_next_index_page_url'>next_index_page_url</span><span class='lparen'>(</span><span class='id identifier rubyid_url'>url</span><span class='comma'>,</span> <span class='id identifier rubyid_pagination_index'>pagination_index</span><span class='rparen'>)</span>
  <span class='kw'>return</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_end'>&#39;</span></span> <span class='kw'>unless</span> <span class='ivar'>@paginated</span>

  <span class='kw'>if</span> <span class='id identifier rubyid_pagination_index'>pagination_index</span> <span class='op'>&gt;</span> <span class='ivar'>@pagination_max_pages</span>
    <span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>&quot;</span><span class='tstring_content'>Exceeded pagination limit of </span><span class='embexpr_beg'>#{</span><span class='ivar'>@pagination_max_pages</span><span class='embexpr_end'>}</span><span class='tstring_end'>&quot;</span></span> <span class='kw'>if</span> <span class='ivar'>@verbose</span>
    <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_end'>&#39;</span></span>
  <span class='kw'>else</span>
    <span class='id identifier rubyid_uri'>uri</span> <span class='op'>=</span> <span class='const'>URI</span><span class='period'>.</span><span class='id identifier rubyid_parse'>parse</span><span class='lparen'>(</span><span class='id identifier rubyid_url'>url</span><span class='rparen'>)</span>
    <span class='id identifier rubyid_query'>query</span> <span class='op'>=</span> <span class='id identifier rubyid_uri'>uri</span><span class='period'>.</span><span class='id identifier rubyid_query'>query</span> <span class='op'>?</span> <span class='const'>Hash</span><span class='lbracket'>[</span><span class='const'>URI</span><span class='period'>.</span><span class='id identifier rubyid_decode_www_form'>decode_www_form</span><span class='lparen'>(</span><span class='id identifier rubyid_uri'>uri</span><span class='period'>.</span><span class='id identifier rubyid_query'>query</span><span class='rparen'>)</span><span class='rbracket'>]</span> <span class='op'>:</span> <span class='lbrace'>{</span><span class='rbrace'>}</span>
    <span class='comment'># update the pagination query string parameter
</span>    <span class='id identifier rubyid_query'>query</span><span class='lbracket'>[</span><span class='ivar'>@pagination_param</span><span class='rbracket'>]</span> <span class='op'>=</span> <span class='id identifier rubyid_pagination_index'>pagination_index</span>
    <span class='id identifier rubyid_uri'>uri</span><span class='period'>.</span><span class='id identifier rubyid_query'>query</span> <span class='op'>=</span> <span class='const'>URI</span><span class='period'>.</span><span class='id identifier rubyid_encode_www_form'>encode_www_form</span><span class='lparen'>(</span><span class='id identifier rubyid_query'>query</span><span class='rparen'>)</span>
    <span class='id identifier rubyid_puts'>puts</span> <span class='tstring'><span class='tstring_beg'>&quot;</span><span class='tstring_content'>Next index pagination url is </span><span class='embexpr_beg'>#{</span><span class='id identifier rubyid_uri'>uri</span><span class='embexpr_end'>}</span><span class='tstring_end'>&quot;</span></span> <span class='kw'>if</span> <span class='ivar'>@verbose</span>
    <span class='id identifier rubyid_uri'>uri</span><span class='period'>.</span><span class='id identifier rubyid_to_s'>to_s</span>
  <span class='kw'>end</span>
<span class='kw'>end</span></pre>
    </td>
  </tr>
</table>
</div>
    
      <div class="method_details ">
  <h3 class="signature " id="next_instance_page_url-instance_method">
  
    - (<tt>Object</tt>) <strong>next_instance_page_url</strong>(url, pagination_index) 
  

  

  
</h3><div class="docstring">
  <div class="discussion">
    
<p>If instance pages are paginated, <strong>you must override</strong>  this
method to return the next URL, given the current URL and its index.</p>

<p>If instance pages aren&#39;t paginated, there&#39;s no need to override
this.</p>

<p>Recursion stops if the fetching URL returns an empty string or an error.</p>

<p>e.g. next_instance_page_url(“<a
href="http://whatever.com/article/upton-sinclairs-the-jungle?page=1">whatever.com/article/upton-sinclairs-the-jungle?page=1</a>”,
2) ought to return “<a
href="http://whatever.com/article/upton-sinclairs-the-jungle?page=2">whatever.com/article/upton-sinclairs-the-jungle?page=2</a>”</p>


  </div>
</div>
<div class="tags">
  

</div><table class="source_code">
  <tr>
    <td>
      <pre class="lines">


130
131
132</pre>
    </td>
    <td>
      <pre class="code"><span class="info file"># File 'lib/upton.rb', line 130</span>

<span class='kw'>def</span> <span class='id identifier rubyid_next_instance_page_url'>next_instance_page_url</span><span class='lparen'>(</span><span class='id identifier rubyid_url'>url</span><span class='comma'>,</span> <span class='id identifier rubyid_pagination_index'>pagination_index</span><span class='rparen'>)</span>
  <span class='tstring'><span class='tstring_beg'>&quot;</span><span class='tstring_end'>&quot;</span></span>
<span class='kw'>end</span></pre>
    </td>
  </tr>
</table>
</div>
    
      <div class="method_details ">
  <h3 class="signature " id="scrape-instance_method">
  
    - (<tt>Object</tt>) <strong>scrape</strong>(&amp;blk) 
  

  

  
</h3><div class="docstring">
  <div class="discussion">
    
<p>This is the main user-facing method for a basic scraper. Call
<code>scrape</code> with a block; this block will be called on  the text of
each instance page, (and optionally, its URL and its index in the list of
instance URLs returned by <code>get_index</code>).</p>


  </div>
</div>
<div class="tags">
  

</div><table class="source_code">
  <tr>
    <td>
      <pre class="lines">


44
45
46
47
48
49</pre>
    </td>
    <td>
      <pre class="code"><span class="info file"># File 'lib/upton.rb', line 44</span>

<span class='kw'>def</span> <span class='id identifier rubyid_scrape'>scrape</span> <span class='op'>&amp;</span><span class='id identifier rubyid_blk'>blk</span>
  <span class='kw'>unless</span> <span class='kw'>self</span><span class='period'>.</span><span class='id identifier rubyid_url_array'>url_array</span>
    <span class='kw'>self</span><span class='period'>.</span><span class='id identifier rubyid_url_array'>url_array</span> <span class='op'>=</span> <span class='kw'>self</span><span class='period'>.</span><span class='id identifier rubyid_get_index'>get_index</span>
  <span class='kw'>end</span>
  <span class='kw'>self</span><span class='period'>.</span><span class='id identifier rubyid_scrape_from_list'>scrape_from_list</span><span class='lparen'>(</span><span class='kw'>self</span><span class='period'>.</span><span class='id identifier rubyid_url_array'>url_array</span><span class='comma'>,</span> <span class='id identifier rubyid_blk'>blk</span><span class='rparen'>)</span>
<span class='kw'>end</span></pre>
    </td>
  </tr>
</table>
</div>
    
      <div class="method_details ">
  <h3 class="signature " id="scrape_to_csv-instance_method">
  
    - (<tt>Object</tt>) <strong>scrape_to_csv</strong>(filename, &amp;blk) 
  

  

  
</h3><div class="docstring">
  <div class="discussion">
    
<p>Writes the scraped result to a CSV at the given filename.</p>


  </div>
</div>
<div class="tags">
  

</div><table class="source_code">
  <tr>
    <td>
      <pre class="lines">


172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189</pre>
    </td>
    <td>
      <pre class="code"><span class="info file"># File 'lib/upton.rb', line 172</span>

<span class='kw'>def</span> <span class='id identifier rubyid_scrape_to_csv'>scrape_to_csv</span> <span class='id identifier rubyid_filename'>filename</span><span class='comma'>,</span> <span class='op'>&amp;</span><span class='id identifier rubyid_blk'>blk</span>
  <span class='id identifier rubyid_require'>require</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>csv</span><span class='tstring_end'>&#39;</span></span>
  <span class='kw'>unless</span> <span class='kw'>self</span><span class='period'>.</span><span class='id identifier rubyid_url_array'>url_array</span>
    <span class='kw'>self</span><span class='period'>.</span><span class='id identifier rubyid_url_array'>url_array</span> <span class='op'>=</span> <span class='kw'>self</span><span class='period'>.</span><span class='id identifier rubyid_get_index'>get_index</span>
  <span class='kw'>end</span>
  <span class='const'>CSV</span><span class='period'>.</span><span class='id identifier rubyid_open'>open</span> <span class='id identifier rubyid_filename'>filename</span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>wb</span><span class='tstring_end'>&#39;</span></span> <span class='kw'>do</span> <span class='op'>|</span><span class='id identifier rubyid_csv'>csv</span><span class='op'>|</span>
    <span class='comment'>#this is a conscious choice: each document is a list of things, either single elements or rows (as lists).
</span>    <span class='kw'>self</span><span class='period'>.</span><span class='id identifier rubyid_scrape_from_list'>scrape_from_list</span><span class='lparen'>(</span><span class='kw'>self</span><span class='period'>.</span><span class='id identifier rubyid_url_array'>url_array</span><span class='comma'>,</span> <span class='id identifier rubyid_blk'>blk</span><span class='rparen'>)</span><span class='period'>.</span><span class='id identifier rubyid_compact'>compact</span><span class='period'>.</span><span class='id identifier rubyid_each'>each</span> <span class='kw'>do</span> <span class='op'>|</span><span class='id identifier rubyid_document'>document</span><span class='op'>|</span> 
      <span class='id identifier rubyid_puts'>puts</span> <span class='id identifier rubyid_document'>document</span><span class='period'>.</span><span class='id identifier rubyid_inspect'>inspect</span>
      <span class='kw'>if</span> <span class='id identifier rubyid_document'>document</span><span class='lbracket'>[</span><span class='int'>0</span><span class='rbracket'>]</span><span class='period'>.</span><span class='id identifier rubyid_respond_to?'>respond_to?</span> <span class='symbol'>:map</span>
        <span class='id identifier rubyid_document'>document</span><span class='period'>.</span><span class='id identifier rubyid_each'>each</span><span class='lbrace'>{</span><span class='op'>|</span><span class='id identifier rubyid_row'>row</span><span class='op'>|</span> <span class='id identifier rubyid_csv'>csv</span> <span class='op'>&lt;&lt;</span> <span class='id identifier rubyid_row'>row</span> <span class='rbrace'>}</span>
      <span class='kw'>else</span>
        <span class='id identifier rubyid_csv'>csv</span> <span class='op'>&lt;&lt;</span> <span class='id identifier rubyid_document'>document</span>
      <span class='kw'>end</span>
    <span class='kw'>end</span>
    <span class='comment'>#self.scrape_from_list(self.url_array, blk).compact.each{|document| csv &lt;&lt; document }
</span>  <span class='kw'>end</span>
<span class='kw'>end</span></pre>
    </td>
  </tr>
</table>
</div>
    
      <div class="method_details ">
  <h3 class="signature " id="scrape_to_tsv-instance_method">
  
    - (<tt>Object</tt>) <strong>scrape_to_tsv</strong>(filename, &amp;blk) 
  

  

  
</h3><table class="source_code">
  <tr>
    <td>
      <pre class="lines">


191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208</pre>
    </td>
    <td>
      <pre class="code"><span class="info file"># File 'lib/upton.rb', line 191</span>

<span class='kw'>def</span> <span class='id identifier rubyid_scrape_to_tsv'>scrape_to_tsv</span> <span class='id identifier rubyid_filename'>filename</span><span class='comma'>,</span> <span class='op'>&amp;</span><span class='id identifier rubyid_blk'>blk</span>
  <span class='id identifier rubyid_require'>require</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>csv</span><span class='tstring_end'>&#39;</span></span>
  <span class='kw'>unless</span> <span class='kw'>self</span><span class='period'>.</span><span class='id identifier rubyid_url_array'>url_array</span>
    <span class='kw'>self</span><span class='period'>.</span><span class='id identifier rubyid_url_array'>url_array</span> <span class='op'>=</span> <span class='kw'>self</span><span class='period'>.</span><span class='id identifier rubyid_get_index'>get_index</span>
  <span class='kw'>end</span>
  <span class='const'>CSV</span><span class='period'>.</span><span class='id identifier rubyid_open'>open</span> <span class='id identifier rubyid_filename'>filename</span><span class='comma'>,</span> <span class='tstring'><span class='tstring_beg'>&#39;</span><span class='tstring_content'>wb</span><span class='tstring_end'>&#39;</span></span><span class='comma'>,</span> <span class='symbol'>:col_sep</span> <span class='op'>=&gt;</span> <span class='tstring'><span class='tstring_beg'>&quot;</span><span class='tstring_content'>\t</span><span class='tstring_end'>&quot;</span></span> <span class='kw'>do</span> <span class='op'>|</span><span class='id identifier rubyid_csv'>csv</span><span class='op'>|</span>
    <span class='comment'>#this is a conscious choice: each document is a list of things, either single elements or rows (as lists).
</span>    <span class='kw'>self</span><span class='period'>.</span><span class='id identifier rubyid_scrape_from_list'>scrape_from_list</span><span class='lparen'>(</span><span class='kw'>self</span><span class='period'>.</span><span class='id identifier rubyid_url_array'>url_array</span><span class='comma'>,</span> <span class='id identifier rubyid_blk'>blk</span><span class='rparen'>)</span><span class='period'>.</span><span class='id identifier rubyid_compact'>compact</span><span class='period'>.</span><span class='id identifier rubyid_each'>each</span> <span class='kw'>do</span> <span class='op'>|</span><span class='id identifier rubyid_document'>document</span><span class='op'>|</span> 
      <span class='id identifier rubyid_puts'>puts</span> <span class='id identifier rubyid_document'>document</span><span class='period'>.</span><span class='id identifier rubyid_inspect'>inspect</span>
      <span class='kw'>if</span> <span class='id identifier rubyid_document'>document</span><span class='lbracket'>[</span><span class='int'>0</span><span class='rbracket'>]</span><span class='period'>.</span><span class='id identifier rubyid_respond_to?'>respond_to?</span> <span class='symbol'>:map</span>
        <span class='id identifier rubyid_document'>document</span><span class='period'>.</span><span class='id identifier rubyid_each'>each</span><span class='lbrace'>{</span><span class='op'>|</span><span class='id identifier rubyid_row'>row</span><span class='op'>|</span> <span class='id identifier rubyid_csv'>csv</span> <span class='op'>&lt;&lt;</span> <span class='id identifier rubyid_row'>row</span> <span class='rbrace'>}</span>
      <span class='kw'>else</span>
        <span class='id identifier rubyid_csv'>csv</span> <span class='op'>&lt;&lt;</span> <span class='id identifier rubyid_document'>document</span>
      <span class='kw'>end</span>
    <span class='kw'>end</span>
    <span class='comment'>#self.scrape_from_list(self.url_array, blk).compact.each{|document| csv &lt;&lt; document }
</span>  <span class='kw'>end</span>
<span class='kw'>end</span></pre>
    </td>
  </tr>
</table>
</div>
    
  </div>

</div>

    <div id="footer">
  Generated on Thu Aug 15 00:25:44 2013 by
  <a href="http://yardoc.org" title="Yay! A Ruby Documentation Tool" target="_parent">yard</a>
  0.8.7 (ruby-2.0.0).
</div>

  </body>
</html>